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INDEPENDENCE  AND  BAYESIAN  UPDATING  METHODS 


I.  BACKGROUND 

Duda,  Hart,  and  Nilsson  [1]  consider  the  problem  of  updating  the  probability  of  a 
hypothesis  H  with  prior  probability  P{H)  when  new  evidence  is  obtained  in  the  form  of 
propositions  Ej  for  which  the  conditional  probabilities  P{Ej  \H)  and  P{Ej  \  H)  are 
known.  They  assume  that  the  Ej  are  conditionally  independent,  both  on  condition  H 
and  on  condition  J7,  so  that 

P  {E  y  •  ■  Em  \H)=  ft  P&j  |  H) 

;=1 

P(Ei -E.  Iff)-  n  P (Ej  Iff). 

They  can  then  use  the  odds-likelihood  form  of  Bayes’s  theorem  and  write  an  updating 
formula  for  the  odds  on  H  in  terms  of  a  product  of  likelihood  ratios: 

P{H\Et---Em)  »  P{E,\H) 

P{R\El  -E„)  P(ff)/i,  P(E,-  |ff)' 

Pednault,  Zucker,  and  Muresan  [2],  in  analyzing  this  updating  scheme,  considered  the 
consequences  of  imposing  the  independence  assumptions  for  each  hypothesis  H{  of  a 
jointly  exhaustive,  mutually  exclusive  set.  They  [2]  and  other  writers  (see  [3]  and  other 
references  therein)  agree  that  the  assumptions  are  unreasonably  strong,  but  there  has 
been  some  confusion  over  the  exact  extent  of  the  undesirable  consequences.  Pednault  et 
al.  [2]  concluded  that  if  there  were  at  least  three  hypotheses,  then  no  updating  could 
take  place — that  the  assumptions  are  too  strong  to  be  satisfied  unless 

P(Ej\H,)  =  P(Ei\Ri)-P(Ei) 

holds  for  all  i  and  j ,  and  consequently 

P(Hi\Et-  -Em)  =  P(Hl). 

However,  Glymour  [3]  gives  a  counterexample  to  their  conclusion— three  jointly  exhaus¬ 
tive,  mutually  exclusive  hypotheses  Hi  and  two  evidence  propositions  Ej  that  satisfy  the 
independence  assumptions  but  allow  updating  to  occur.  He  points  out  that  Pednault  et 
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al.  had  relied  on  an  erroneous  result  claimed  by  Hussain  [4],  also  refuted  by  his  coun- 
terexample.  Glymour  notes  that  the  evidence  proposition  E2  of  his  counterexample  has 
the  special  property  that  P(E2  \  Hi)  =  P{E2  |  H 3)  =  0,  so  that  E2  determines  a  poste¬ 
rior  probability  of  1  for  one  hypothesis,  Hlt  and  0  for  the  rest.  He  raises,  and  leaves 
open,  the  question  whether  Pednault  et  al' s  result  would  be  true  with  the  additional 
requirement  that  for  all  * , 


m- 


Em)^  0. 


In  the  next  section  we  answer  that  question  by  giving  a  counterexample  that 
satisfies  (1). 

Glymour’s  counterexample  has  another  special  property,  one  that  is  sufficient  to 
make  it  a  valid  counterexample:  P{Ex  |  H{)  =  P(EX  |  Si)  =  P(EX),  so  that  Ex  pro¬ 
duces  no  updating;  only  E2  produces  updating.  We  give  a  second  counterexample  that 
lacks  this  second  special  property.  However,  we  can  show  that  Pednault  et  al.’ s  assump¬ 
tions  imply  that  for  every  t  there  is  at  most  one  value  of  j  for  which 

does  not  hold.  We  show  this  in  the  section  following  the  counterexamples.  It  follows 
that  for  each  hypothesis  Hx ,  there  is  at  most  one  evidence  proposition  Ej  that  produces 
updating  of  the  probability  of  Ht ;  there  is  no  hypothesis  for  which  multiple  updating  is 
possible. 

We  conclude  this  section  by  stating  the  assumptions  used  by  Pednault  et  al.  The 
hypotheses  are  Hx,  .  .  .  ,  Hn  ;  it  is  assumed  that 

n  >  2  (2) 

and  that  the  //,  are  jointly  exhaustive 

£m)  =  i  (3) 

»-i 

and  mutually  exclusive 

P (.H,  Hj )  ==  0  (4) 

The  evidence  propositions  are  Elt  ...  ,  Em .  Since  a  subset  of  an  independent  set  is 
independent,  we  write  the  independence  assumptions  as 

P{Ejx  -  Ejk\Hi)^l[P{Ej\Hi),  (5) 

P(Eh  --Eh\Ei)^Y[P{Ej\Hi).  (6) 

for  every  subset  /  =  {jv  .  .  .  ,  jk  }  of  the  indices  1,  .  .  .  ,  m . 
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II.  COUNTEREXAMPLES 


For  comparison,  here  is  Glymour’s  counterexample. 

{Ll  mi Mil 

H i  1/6  0  1/6  0 

H2  0  1/6  0  1/6 

Hz  0  1/6  0  1/6 

It  is  straightforward  to  verify  that 

P(E 2  I  Hx)  =  1, 

P(E.\H2)  =  P(Et\H%)  =  0, 
as  noted  by  Glymour,  and  that 

P(E,\Hi)  =  P(El\ni)=P(El) 

for  each  i . 

To  answer  the  question  raised  by  Glymour,  we  modify  the  example  so  that  (7)  no 
longer  holds.  We  retain  the  same  values  for  P  (Hi )  and  P(Ei\  Hj ),  choose  new,  nonzero 
values  for  P(E2\Hi)  (say  1/2,  1/3,  1/6  for  *  =1,2,3),  and  define  the  remaining 
relevant  conditional  probabilities  and  probabilities  with  the  help  of  (5).  Here  is  the 
result. 

ik  PiEiEtHj)  P(EXE2H{)  P(ExE2Hj)  P(EXE2H{) 


H  x 

1/12 

1/12 

1/12 

1/12 

h2 

1/18 

1/9 

1/18 

1/9 

h3 

1/36 

5/36 

1/36 

5/36 

Assumptions  (2)-{6)  can  be  verified.  In  fact,  we  can  show  that  as  long  as  E  x  satisfies  (8), 
we  can  choose  P(E 2  |  Hi)  arbitrarily,  and  the  procedure  we  have  just  used  will  lead  to  a 
probability  distribution  that  satisfies  (2)-{6). 

Now  (8)  implies  that  Ex  is  irrelevant  for  inference  about  the  hypotheses.  Only  E2 
produces  updating— multiple  updating  does  not  occur.  But  by  going  to  four  hypotheses, 
we  can  dispense  with  (8)  and  obtain  a  counterexample  such  that  Ex  and  E 2  can  both 
produce  updating. 


Hl 

P(EXE2H) 

P(ExE2Hi ) 

p(Exe* 

Hi)  P(ExEoHi) 

H  x 

1/24 

1/24 

1/12 

1/12 

h2 

1/12 

1/12 

1/24 

1/24 

h3 

1/24 

1/12 

1/24 

1/12 

h4 

1/12 

1/24 

1/12 

1/24 

Again  (2)-(6)  can  be  verified.  Furthermore  Ex  and  E2  can  both  produce  updating  since 


(7) 

(8) 


we  have,  for  example,  P(EX  |  Hx)  P(E i)  and  P(E2\H3)j^  P(E 2).  However,  we 
have 

P(E,  |  »,)  =  />(£■,  |  £•)  =  />(£.)  (.  =3.4), 

P(E,\Hi)~P(E2\Bi)  =  P (£,)  (i  =l.  2). 

Thus  only  Ex  can  update  the  probability  of  Hi  or  H2,  and  only  E2  can  update  the  pro¬ 
bability  of  H3  or  H for  no  hypothesis  is  multiple  updating  possible.  This  illustrates 
the  general  case,  as  we  show  in  the  next  section. 

HI.  IMPOSSIBILITY  OF  MULTIPLE  UPDATING 

Theorem.  If  the  assumptions  (2 )-(6)  hold,  then  for  every  H{  there  is  at  most  one 
Ej  that  produces  updating  for  //, . 

Proof.  No  updating  for  Hi  is  possible  if  either  P  ( Hi )  or  P  (Hi )  is  0;  therefore  we 
may  assume  that  both  are  nonzero.  First  consider  the  case  m  —  2,  where  the  evidence 
propositions  are  just  E\  and  E2.  We  follow  Pednault  et  al.  (see  [2],  equations  (6)— {9})  in 
deriving 

P(El)P(E2)  -  P{Ei)P[E2Hi)  -  P(EiHi)P(E2) 

=  PiElE2)ll-P(Hi))-P{EtE2Hi)  ” 

and  summing  over  i  to  obtain 

(n  -2)P(Ei)P(E2)  =  (n  -  2)P(ElE2), 
or 

P(Ex)P{E2)  =  P(EiE2).  (10) 

Using  (9)  and  (10)  with  the  help  of  (5),  we  obtain 

P  (E  i)P  (E2\Hi)P{Hi)  +  P(Ex\  ^  )P  (E2)P  (H{ ) 

=  P  [E{)P  (E2)P  (Hi )  +  P  (Ex  |  ^  )P  (E2  |  Hi  )P  (Hi ) 
from  which  it  follows  that 

\P(Ex)  -  P (Ex  |  Hi))[P(E2)  -P(E2  |  Hi))  =  0. 

One  of  the  bracketed  factors  vanishes.  If  P(Ej  |  //,  )  =  P(Ej),  then  in  fact 

p(Ej  \Hi)  =  p(Ej\ni)  =  p(Ejy, 

this  therefore  holds  either  with  j  =  1  or  with  j  =  2.  Consequently  Ex  and  E2  do  not 
both  produce  updating  for  Hi .  Thus  we  have  proved  the  theorem  for  the  case  m  —  2. 
But  in  the  general  case,  if  one  of  the  evidence  propositions,  say  Ej ,  produces  updating 
for  Hi ,  the  result  for  m  =  2  implies  that  no  other  evidence  proposition  Eh  produces 
updating  for  H{ . 
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IV.  DISCUSSION 


Assumptions  (2)-(6)  lead  to  unreasonably  severe  restrictions  on  the  possibility  of 
probabilistic  updating.  If  we  wish  to  make  inferences  about  more  than  two  jointly 
exhaustive,  mutually  exclusive  hypotheses,  and  if  we  wish  to  allow  more  than  one  piece 
of  evidence  to  bear  on  one  hypothesis,  then  we  must  eliminate  either  (5)  or  (6).  Duda  et 
al.  made  the  assumptions  (5)  and  (6)  in  the  context  of  a  single  pair  of  hypotheses  H ,  H . 
It  is  clearly  a  mistake  to  carry  them  both  over  into  the  context  of  several  exhaustive, 
mutually  exclusive  hypotheses. 

Cases  where  (5)  is  justified  (at  least  as  an  approximation)  are  quite  common,  but 
(6)  is  much  less  plausible.  Consider,  for  example,  a  physical  quantity  x  that  can  take 
any  of  a  considerable  range  of  numerical  values  t >,• .  Let  y  and  z  be  measurements  of  x 
made  with  instruments  subject  to  independent  errors;  that  is,  suppose  y  -  x  and  z  -  x 
are  independent  random  variables  but  are  fairly  small  with  high  probability.  Then 
y  -  z  is  small  with  high  probability,  and  so  y  and  z  are  highly  dependent;  but  on  con¬ 
dition  of  a  given  value  of  x ,  say  x  =  t/,- ,  the  conditional  distributions  of  y  and  z  are 
independent.  Define  Hi  to  be  x  =  v,- .  Then  y  and  z  are  independent  on  condition  Hi 
for  any  i .  However,  on  condition  H{ ,  we  expect  them  to  be  dependent,  for  the  same 
reason  that  their  unconditional  distributions  are  dependent.  In  that  case,  we  can  take 
Ex  and  E 2  to  be  propositions  about  y  and  z,  respectively,  and  it  is  easy  to  choose  E x 
and  Ez  so  that  they  are  conditionally  independent  on  condition  Ht  but  not  on  condition 
Hi. 

We  must  conclude  that  for  inference  about  jointly  exhaustive,  mutually  exclusive 
hypotheses,  updating  schemes  based  on  the  independence  assumption  (5)  alone  may  be 
useful,  but  schemes  based  on  both  (5)  and  (6)  are  too  restrictive  to  be  useful. 
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